STEREOSCOPIC PANORAMIC IMAGE CAPTURE DEVICE 



Background of the Invention 

Field of the Invention 

The present invention relates in general to a panoramic image capture device and, more 
specifically, to a panoramic image capture device for producing a stereoscopic panoramic image. 

Description of the Prior Art 

Panoramic cameras are known in the art. Such cameras often use a single, rotatable 
camera. Although such devices are suitable for stationary images, such devices typically 
produce blurred or distorted images when used to capture non- stationary objects. It is also 
known in the art to utilize an image capture system having a plurality of image capture devices. 
In this manner, a plurality of images can be captured, substantially simultaneously, and stitched 
together using processes known in the art. Although such systems substantially eliminate the 
problem associated with capturing objects in motion, such systems do not provide means for 
producing a stereoscopic image. 

It is also known in the art to use a "fish-eye" lens to capture a panoramic image. Such 
images, however, import a large amount of distortion into the resulting image, and capture 
images of relatively low quality. Accordingly, it would be desirable to produce a panoramic 
image of lower distortion and higher quality. 

Typically, to capture a stereoscopic image, two imaging systems are positioned next to 
one another to capture a particular image. Unfortunately, this method cannot be extrapolated to 
producing a stereoscopic panoramic image, as one image capture device would necessarily fall 
into the field of view of the adjacent image capture device. It would be desirable, therefore, to 
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provide a panoramic image capture system which could be utilized to produce a stereoscopic pair 
of panoramic images for a stereoscopic display of a particular image. 



Summary of the Invention 

In an advantage provided by this invention, an image capture system produces a 
stereoscopic pair of panoramic images. 

Advantageously, this invention provides an image capture system for producing a 
seamless panoramic image. 

Advantageously, this invention provides an image capture system for producing a 
stereoscopic panoramic image in motion. 

Advantageously, this invention provides a stereoscopic panoramic image of minimal 
distortion. " ------ 

Advantageously, this invention provides an imaging system for full-motion, real time, 
panoramic stereoscopic imaging. 

Advantageously, in a preferred example of this invention, an imaging system is provided 
comprising a first image capture device, a second image capture device, and a third image 
capture device. Means are also provided for combining at least a first portion of a first image 
captured using the first image capture device with a portion of a second image captured using the 
second image capture device to produce a first combined image. Means are also provided for 
combining at least a second portion of the first image with at least a portion of a third image 
captured using the third image capture device, to produce a second combined image. 

In the preferred embodiment, a plurality of image capture devices are utilized to produce 
a plurality of images, a portion of each of which is combined with a portion of adjacent images 
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to produce a first combined panoramic image. Similarly, a second portion of each image is 
combined with separate portions of adjacent images to produce a second combined panoramic 
image. Preferably, the first combined panoramic image, and second combined panoramic image 
are displayed in a stereoscopic orientation to produce a stereoscopic panoramic image. The 
imaging system of the present invention may be utilized to capture a plurality of images, to 
produce a full-motion stereoscopic panoramic image. 

Brief Description of the Drawings 

The present invention will now be described, by way of example, with reference to the 
accompanying drawings in which: 

Fig. 1 illustrates a front elevation of the imaging system of the present invention; 

Fig; 2 illustrates a graphic depiction of image capture regions of adjacent image capture 
devices of the imaging system of the present invention; 

Fig. 3 illustrates a bottom perspective view of the final panoramic stereoscopic image 
displayed on a screen, and the polarized glasses used to view the image; 

Figs. 4A-4B illustrate a left panoramic image and right panoramic image; and 

Figs. 5A-5B illustrates images associated with first image buffer and second image 

buffer; 

Figs. 6A-6C are a flowchart of the transformation process utilized in association with the 
imaging system of the present device, to transform the plurality of images captured with the 
imaging devices into a first panoramic image and a second panoramic image to produce a 
stereoscopic panoramic image; and 
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Fig. 7 illustrates a perspective view of a 360 degree unobstructed panoramic stereoscopic 
image, created using the camera of Fig. 1 . 



Detailed Description of the Preferred Embodiment 

Referring to Fig. 1, a camera (10) is shown having a body (12) constructed of plastic or 
other similar lightweight material. In the preferred embodiment, the body (12) is substantially 
spherical, having a diameter preferably between .001 and 500 centimeters, and more preferably, 
between 10 and 50 centimeters. Provided substantially equally spaced across the surface of the 
body (12) are a plurality of lenses (14). The lenses (14) are preferably circular lenses having a 
diameter of preferably between 5 angstroms and 10 centimeters, and more preferably, between .5 
and 5 centimeters. In the preferred embodiment, the lenses are model number BCL38C 3.8 
millimeter micro lenses, manufactured by CBC America, located at 55 Mall Drive, Commack, 
NY 1 1725. As shown in Fig. 2, the lenses (14) are each associated with a charge coupled device 
(CCD) assembly (16), such as those well known in the art. Although in the preferred 
embodiment a GP-CX171/LM CCD color board camera, manufactured by Panasonic and 
available from Rock2000.com is used, any known image capture system may be used. As shown 
in Fig. 2, all of the CCD assemblies (16) are coupled to a central processing unit (CPU)(22). In 
the preferred embodiment, the CPU (22) is a 900 MHz, Pentium®4 class personal computer 
provided with an Oxygen GVX210 graphics card manufactured by 3Dlabs of 480 Pontrero, 
Sunnyvale, CA 94086. Although the CPU may be of any type known in the art, it is preferably 
capable of quad buffering and utilizing page flipping software in a manner such as that known in 
the art. The CPU (22) is coupled to a head mounted display (24) which, in the preferred 
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embodiment, is a VFX3D, manufactured by Interactive Imaging Systems, Inc., located at 2166 
Brighton Henrietta Townline Road, Rochester, NY 14623. 

As shown in Fig. 2, the lenses (14) are divergent from one another, and offset twenty 
degrees from one another along the arcuate surface of the body (12). This divergency allows 
each lens to have a substantially similar focal point. Each lens (14) has a fifty-three degree field 
of view, which overlaps the field of view of a laterally adjoining lens by between ten and ninety 
percent, and, preferably, between fifty and sixty-five percent. As shown in Fig. 2, a first image 
capture unit (26) is associated with an optical axis (28), bisecting the image bordered on one side 
by a left side plane (30) and on the other side by a right side plane (32). The lens (14) of the 
image capture unit (26) is focused on a defined image plane (34), divided into a left image plane 
(36) and a right image plane (38). 

Similarly, a second image capture unit (40)_is also provided with an optical axis (42), a 
left side plane (44), a right side plane (46), a defined image plane (48), a left image plane (50), 
and a right image plane (52). A third image capture unit (54), to the right of the first image 
capture unit (26), is provided with optical axis (56), a left side plane (58), a right side plane (60), 
a defined image plane (62), a left image plane (64), and a right image plane (66). 

By providing a plurality of image capture units (26), (40) and (54), dividing the defined 
image planes (34), (48) and (62) in halves, and orienting the image capture units (26), (40) and 
(54) as shown in Fig. 2, every point associated with a final panoramic image is within the 
defined image plane of at least two adjacent image capture units, (26), (40) or (54). As shown in 
Fig. 5A-5B, the defined image planes of adjacent image capture units overlap vertically, 
preferably about 1-20 percent, more preferably 5-10 percent, and 7 percent in the preferred 
embodiment. Similarly, the defined image planes of adjacent image capture units overlap 



horizontally, preferably about 20 percent, more preferably 5-10 percent, and 6 percent in the 
preferred embodiment. These overlaps aid in the "stitching" process described below. 

To produce the final panoramic image (68) of the present invention, a first panoramic 
image (72) and second panoramic image (74) are created. (Figs. 3 and 4A-4B). To create the 
first panoramic image (72), an image (76), associated with the left image plane (36) of the first 
image capture unit (26) is combined with an image (78), associated with the left image plane 
(50) of the second image capture unit (40) and an image (80), associated with the left image 
plane (64) of the third image capture unit (54). (Figs. 2, 4A and 5 A). As shown in Fig. 2, the 
associated image planes (36), (50) and (64), preferably overlap by .5-30 percent, more preferably 
by 10-20 percent, and 13 percent in the preferred embodiment, but are not parallel to one 
another, and are not tangent to a curve defining the first panoramic image (72). (Fig. 2 and 4A). 
- Accordingly, the images (76), (78), (80), (82), (84) and (86), associated with planes (36), (50) 
and (64) must be transformed, to remove distortion associated with their non-parallel orientation, 
before they can be stitched together as described below to form the final panoramic image (68). 
(Figs. 2, 3, 4A-4B and 5A-5B). 

Once the images (76), (78) and (80), associated with the left image planes (38), (52), 
(66), and the images (82), (84) and (86), associated with the right image planes (36), (50) and 
(64) have been collected from all of the image capture units (26), (40) and (54), the images are 
transmitted via hardwired, wireless, or any desired connection to the CPU (22). The CPU (22) 
then transforms the images in accordance with the process described in Figs. 6A-6C. (Figs. 2, 4A 
and 5 A). As shown in step (88), source images, which in the preferred embodiment are 
rectilinear images, but which may, of course, be any type of image, are obtained from the image 



capture units (26), (40) and (54). (Figs. 2, 4A, 4B and 6). As shown in step (94), the CPU (22) 
defines registration pixel pairs for the untransformed source images. 

Thereafter, as shown in step (96), the CPU(22) creates an input file. The input file 
includes the height and width of the final panoramic image (68), the source information, and 
registration point information. The source information includes the file name and path of the 
source image, the height and width of the source image in pixels, and the yaw, pitch and roll 
angles of the source of the associated image capture unit. The horizontal field of view of the 
image capture unit, which is preferably between 1 and 80 degrees, more preferably between 30 
and 60 degrees, and 53 degrees in the preferred embodiment, is defined by the left side plane and 
right side plane, the X Shift, Y Shift and zoom values of the source image. The information 
associated with the registration points includes information regarding the source image 
associated with the first pixel position of the registration point, a horizontal and vertical pixel 
position in the first source image, list index of information regarding the source image associated 
with the second pixel position, and a horizontal and vertical pixel position in the second source 
image. 

The images (76-86) associated with the image planes (36, 38, 50, 52, 64 and 66) are 
rectilinear, normal flat-field images, and the panoramic images (72 and 74) are equirectangular, 
representing pixel mapping on a spherical surface in a manner such as that shown in Figs. 4A- 
4B. Accordingly, once the CPU (22) creates the input file, as shown in step (98), the registration 
pixel pairs must be transformed to locate their position in the final panoramic image (68). 

Starting with an arbitrary source image, a vector is defined that represents a first pixel of 
a given registration pixel pair in three-dimensional space, locating it on the final 



panoramic image (68). This is accomplished by applying the following matrix 
transformation to each pixel: 

Define segment x: Alter the horizontal position of the pixel in the source image so that it 
is relative to the image's center. Then compensate for the xShift and zoom variables of 
the source image. 

Define segment y: Alter the vertical position of the pixel in a similar way. 

Define segment z: Using various source image variables, determine the z segment that 
corresponds to the scale provided by the image's size in pixels. 

Transform the vector so that it corresponds to the rotation angles of the source camera. 
This is calculated by transforming the vector by each of three rotational matrices: 

• Rotation about x-axis: 

r \ 0 0 0 > 

0 cos(pitch) sin(pitch) 0 
0 - $in( pitch) cos(piich) 0 
^0 0 0 1; 

• Rotation about y-axis: 
r cos(yaw) 0 -sm(yaw) 0^ 

0 10 0 

sin(j>avi>) 0 cosQ^ow) 0 

^ 0 0 0 \j 

• Rotation about z-axis: 

f cos(ro//) sin(ra//) 0 0^ 

-sin(ro//) cos(ro//) 0 0 
0 0 10 

K 0 0 0 1; 

Upon matrix transformation, the pixel vector represents the global globX, globY, and globZ 
positions of that point in three-dimensional space. The CPU (22) then converts these positions 
into spherical coordinates and applies them directly to the final panoramic coordinates. The 



8 



vector's yaw angle represents its horizontal panorama position, newX, and its pitch angle 
represents its vertical panorama position, newY. 

As shown in step (100), once the registration pixel pairs have been mapped to the final 
panoramic image (68) in this manner, the CPU (22) calculates the distance between the 
registration pixel pairs. If the average distance of the registration pixel pairs for a given source 
image are not yet minimized, as would be the case upon the initial transformation, shown in step 
(102), the yaw of the source image is altered slightly, whereafter the process returns to step (98) 
and the registration pixel pairs are again transformed to pixel points in the final panoramic image 
(68). This process continues, altering the yaw, until the average distance of the source image 
registration pixel pairs is minimized. Thereafter, the pitch, roll, X Shift, Y Shift and zoom are 
altered, until the average distance of the associated registration pixel pairs is minimized. Once 
the yaw, pitch, roll, X Shift, Y Shift and zoom of a particular source image is optimized, as 
shown in step (104), the transformation procedure is repeated for all source images until they are 
all thus optimized. 

As shown in step (106), once all of the source images have been thus optimized, the 
average distance of all source image registration pixel pairs is calculated and, if they are not yet 
minimized, the yaw, pitch, roll, XShift, YShift and zoom of the source images are altered as 
shown in step (108), and the process returns to step (98), where the process continues, until the 
distance between the registration pixel pairs is minimized across all of the source images. 

Once the average distance between the registration pixel pairs has been minimized across 
all source images, as shown in step (110), an output file is created, identifying the height and 
width of the first panoramic image (72), the yaw, pitch, roll, XShift, YShift and zoom 
transformation image information relative to each particular source image. Thereafter, as shown 



in step (1 12), for each pixel within a given source image, a vector is defined representing the 
particular pixel's position in three dimensional space, using the vector transformation described 
above. 

Once a vector has been defined, as shown in step (114), the vector is transformed to 
reflect the yaw, pitch, roll, XShift, YShift and zoom information associated with the source 
image as defined in the output file. After completion of the transformation of the vector, as 
shown in step (116), the transformed vector is associated with a pixel in the final panoramic 
image (68). As shown in step (118), this process is repeated until all of the pixels in a particular 
source image have been transformed into vectors, and their position located on the final 
panoramic image (68) 

As shown in step (120), once all of the pixels of a given source image have been located 
on the final panoramic image (68), two image buffers (90) and (92) are created, each having a 
height and width equal to that of the final panoramic image (68). (Figs. 3, 5A-5B and 6B). As 
shown in step (122), once the image buffers (90) and (92) have been created, vector 
transformation information associated with a quadrilateral of four adjacent pixels of a particular 
source image is utilized to draw the quadrilateral of pixels onto the appropriate image buffer (90) 
or (92). (Figs. 5A-5B and 6C). If the pixel is in the left image planes (38), (52) or (66), the pixel 
is written to the left image buffer (90). If the pixel is in the right image planes (36), (50) or (64), 
the pixel is in the right image buffer (92). (Figs. 2 and 5 A-5B). 

As shown in step (124), since the transformation is likely to spread out the quadrilateral 
of pixels on the image buffer, there will likely be gaps between pixels as they are converted from 
their rectilinear location to their equirectangular location in the associated image buffer (90) or 
(92). (Figs. 5A-5B and 6C). When the quadrilateral of pixels is located on the associated image 
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buffer (90) or (92), the gaps thereby created are filled as smoothly as possible, using a linear 
gradient of the corner pixel colors, in a manner such as that known in the art. Known linear 
gradient fill techniques are also used to diminish visible seams between images. As additional 
source image information is applied to the image buffers (90) and (92), for areas of the image 
buffers (90) and (92) that have already been filled with pixel data, the alpha transparency of the 
new overlapping pixels is linearly degraded, smoothing the resulting seam as described below. 

Accordingly, when mapping the quadrilateral to the associated image buffer (90) or (92), 
the central processing unit (22) eliminates the gaps by interpolating the internal pixel values, 
using any method known in the art, which may include comparing the gap to the adjacent pixels, 
or "white boxing" the gap by utilizing the immediately preceding frame and immediately 
succeeding frame in a motion capture system, to extrapolate the most appropriate pixel value for 
the gap. As shown in step (126), steps (122) and (124) are repeated until all of the source image 
pixels have been mapped to the appropriate image buffer (90) or (92). 

Once all of the pixels have been mapped, as shown in step (128), the first image buffer 
pixels are compared to the first panoramic image pixels. If the first panoramic image (72) does 
not have a pixel associated with a pixel in the first image buffer (90), the CPU (22) sets the pixel 
in the first image buffer (90) to maximum visibility. If the first panoramic image (72) already 
has a pixel associated with a pixel in the first image buffer (90), the existing pixel is compared to 
the corresponding pixel in the first image buffer (90). If the pixel in the first image buffer (90) 
has a shorter distance to the center of its respective source image than does the existing pixel, the 
pixel in the first image buffer (90) is set to maximum visibility. Conversely, if the pixel in the 
first image buffer (90) has a greater distance to the center of its respective source image than that 
of the existing pixel, then the pixel in the first image buffer (90) is set to minimum visibility. 
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The process is repeated to merge pixels of the second image buffer (92) into the second 
panoramic image (74). 

As shown in step (130), the overlapping edges of the image in the image buffers (90) and 
(92) are feathered by degrading visibility of the pixels over the area of overlap. This feathering 
smoothes the overlapping areas of the image once it is merged from the image buffers (90) and 
(92) into the panoramic images (72) and (74). As shown in step (132), once the image buffer 
pixels have been set to the appropriate visibility, and the image of the image buffers feathered, 
the images in the image buffers are merged into the first and second panoramic images (72) and 
(74). As shown in step (134), steps (122) through (132) are repeated until all of the source 
images have been merged into the first and second panoramic images (72) and (74). As noted 
above, the images (76), (78) and (80) associated with the left image planes (36), (50) and (64) of 
the image capture units (26), (40)" and (54) are used to create the first panoramic image (72), and 
the images (82), (84) and (86) associated with the right image planes (38), (52) and (66) of the 
image capture units (26), (40) and (54) are used to create the second panoramic image (74). 

Once both of the final panoramic images (72) and (74) have been created, as shown in 
step (138), the panoramic images (72) and (74) are displayed together as the final panoramic 
image (68). (Figs. 3, 4A and 4B). The panoramic images (72) and (74) may be reverse polarized, 
shown on a standard panoramic screen (140), such as that shown in Fig. 4, and viewed using 
glasses (142) having lenses of reversed polarity. Alternatively, the panoramic images (72) and 
(74) may be conveyed digitally to the head mounted display (24) shown in Fig. 1. The CPU (22) 
can send multiple panoramic images to the head mounted display (24) using known "page- 
flipping" techniques to send and separately display the images in the left and right displays of the 
head mounted display (24), as appropriate. This process can be used to animate the display as a 
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full 24 frame per second motion picture, or to display multiple visual images to each display. 
The head mounted display unit (24) may also be provided with an orientation sensor (144), such 
as those well known in the art, to change the image provided to the head mounted display (24) as 
the imaging unit moves. In this manner, a user (not shown) may look up, down, and in either 
direction, to see that portion of the final panoramic image (68) associated with the vector of the 
user f s line of sight. In this manner, the user will have the feeling of actually looking around a 
three-dimensional space. 

In an alternative embodiment of the present invention, the camera may be provided with 
a plurality of image capture pairs having rectilinear capture systems oriented substantially 
parallel to one another. The pairs are offset by a predetermined factor to obtain a desired 
stereoscopic image. Because the images are captured in pairs, the transformation process 
associated with this embodiment is identical to that described above, albeit instead of dividing 
the images in half, and sending pixels from each half to a separate image buffer, all of the pixels 
associated with the images from the "left" image capture devices of each image capture device 
pair are sent to one image buffer, and all of the pixels associated with images from the "right" 
image capture devices are sent to another image buffer. 

In another alternative embodiment of the present invention, the camera (10) and CPU 
(22) may be utilized to capture twenty-four or more frames per second, and display the final 
panoramic image (68), in real time, as a motion picture. 

In yet another alternative embodiment of the present invention, computer generated 
graphical information, produced in a manner such as that well known in the art, may be 
combined with the final panoramic images (72) and (74) in the central processing unit (22) to 
provide a seamless integration of actual images captured with the camera (10), and virtual reality 
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images (146). (Fig. 3) This combination produces a seamless display of real and virtual 
panoramic stereographic virtual images. 

In yet another alternative embodiment of the present invention, the images captured by 
the camera (10) may be transformed, utilizing the above transformation procedures, to produce a 
seamless 360-degree panoramic monographic image. As shown in Fig. 1, the camera (10) is 
provided with a support post (148) and a transportation unit such as a remote control carriage 
(150) used in association with remote control cars and the like. Images associated with the left 
image planes of the image capture units may be used to produce a combined image and the 
images associated with the right image planes of the image capture units may be used to 
overwrite and fill the combined image to hide the support post (148), carriage (150) and any 
other camera equipment otherwise visible in the combined image. The foregoing transformation 
procedures may be utilized for such overwriting and filling. In this manner, only those images 
which do not include the undesired information are mapped to a final panoramic image (152), 
which may be displayed on a spherical screen (154) or on the head mounted display (24). (Figs. 
1 and 7). For that portion of the image reflecting the area covered by the footprint of the carriage 
(150), the interpolation and feathering procedures detailed above may be utilized to approximate 
the image lying beneath the carriage (150), to produce the appearance of a completely 
unobstructed 360-degree image. 

Although the invention has been described with respect to the preferred embodiment 
thereof, it is to be also understood that it is not to be so limited, since changes and modifications 
can be made therein which are within the full intended scope of this invention as defined by the 
appended claims. 
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